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ABSTRACT 



A study investigated the concurrent validity of the cloze 
test administered with an essay test to Korean university students. Subjects 
were 129 students enrolled in Engl ish-as -a- Second- Language courses in Korea. 

A seventh-word deleted cloze test and a descriptive essay test were 
administered. Analysis of results indicate a significant correlation between 
the two tests, confirming the findings of two previous studies that cloze 
tests can be used as an alternative to essay tests. Theoretically, the 
results suggest that writing proficiency may be an important factor in cloze 
performance. Practically, the results suggest that the cloze procedure may be 
effective as a teaching device. The cloze test, cloze test instructions, and 
essay test instructions used are appended. Contains 21 references. 
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The Concurrent Validity of Cloze Test with Essay Test among 
Korean Students 

SOYOUNG LEE 

This study investigated the concurrent validity of the cloze test with the 
essay test among Korean university students. A total of 129 students 
enrolled in freshmen English courses in Seoul, Korea, participated in 
this study. A seventh-word deleted cloze test and a descriptive essay test 
were administered to the students. A significant correlation between the 
two tests (r=.67 by the exact-word method and .64 by the acceptable- 
word method) was found. This result confirms the finding of two 
previous studies (Fotos, 1991; Hanania and Shikhani, 1986) that cloze 
tests can be an alternative to essay tests. This study has both theoretical 
and practical implications. Theoretically, the finding of this study 
seems to be an answer to the question of the construct of the cloze test. 
Practically, it suggests the use of the cloze procedure as a teaching 
device in classroom situations in Korea. 

INTRODUCTION 

Discrete-point tests have been widely used in evaluating students' English 
language proficiency in Korea. The advantages of discrete-point tests such as 
machine grading that enables to control massive number of examinees have been 
appealing. However, important concerns have been expressed in the second and 
foreign language literature that performance on a discrete-point test may indicate 
not overall language ability but intensive learning of isolated grammatical rules 
(Hanania and Shikhani, 1986). The same concerns have been also a critical issue 
in the English education field in Korea. The situation has been worrisome; six 
years of English learning during the period of middle to high schools results in 
only some reading ability but little ability of speaking, listening, and writing for 
most students. It is understandable considering that most English teachers in 
Korea are nonnative speakers and that, unlike listening and reading tests, it is 
difficult to develop objective scoring formats for speaking and writing tests. 
However, the disadvantages of a discrete-point test for the evaluation of the 
productive ability merit serious concern; such a test tells a great deal more about 
learners' ability to recognize correct elements of the language than about their 
ability to use their knowledge effectively with a listener or a reader. 

Accordingly, the use of an integrative test for the evaluation of speaking and 
writing ability deserves attention. In particular, if there is an integrative test that 
is easy to prepare, administer, and score in evaluating the productive skills of 
speaking and writing, it will be very useful for English teachers in Korea who 
experience difficulties in dealing with the productive ability on tests. In this 
respect, cloze tests seem to be a promising approach. In fact, the usefulness of 
cloze tests has been recognized by some researchers. Hanania and Shikhani 
(1986) found significant correlations among a cloze, an essay, and a standard 
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ESL tests. Fotos (1991) reported that with careful preparation the use of cloze 
test as a writing proficiency measure can yield high validity and reliability 
estimates. 

This study has a purpose of confirming the concurrent validity of the cloze test 
with the essay test among Korean students. So far, many studies have attempted 
to validate cloze tests with various discrete-point tests (Bachman, 1985; Brown, 
1980, 1983) and shown a significant correlation between cloze tests and discrete- 
point tests. However, there has been little research on the validation of cloze 
tests with integrative tests to which cloze tests belong. Shohamy's (1983) study 
investigating the relationship between an oral interview and the cloze procedure, 
and Fotos’s (1991) and Hanania and Shikhani’s (1986) studies demonstrating 
high validity of cloze tests with compositions are a few studies. Thus, this study 
is to see whether cloze tests can measure the ability that essay tests measure 
among Korean students. 

METHODOLOGY 

Subjects 

The subjects in this study were 129 college first-year Korean students who 
were native speakers of Korean studying in two private universities in Seoul, 
Korea in March, 1995. Their majors were electronics, child and family, sport and 
leisure studies, Russian language and literature, chemical engineering, and 
management. Eighty-five students were male and forty-four students were female. 
They were enrolled in freshmen English classes that are required for all freshmen 
in the universities. 

The subjects in this study were a homogeneous group with regard to 
nationality, language background, age, and educational level. The students ranged 
in age from 18 to 20. While no measure of overall English language proficiency 
was available for the students, it can be said that all of the students had taken six 
years of formal English courses in middle and high schools prior to this study. 
Their 1995 college entrance English test scores indicated they were in the top 
10% group of those who took the test. 

Material 

This study needed two kinds of material: a cloze test and an essay test. The 
cloze test was prepared according to the procedures of Hanania and Shikhani 
(1986) and Fotos (1991). First, the students' reading ability level that would 
guide the choice of cloze texts was decided by Dale and Chall's (1948) readability 
formula using two college entrance English tests. Then, three different texts were 
picked from two textbooks designed for seventh or eighth graders in the U.S.A. 
Finally, three passages of about three hundred words for the rate of fifth word 
deletion were prepared to yield fifty blanks from the texts. From the passages, 
three different cloze tests were constructed. A pilot study led to revised cloze tests 
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with seventh deletion rate, thirty blanks, and more contextual information. As a 
pretest, the revised versions of the three cloze tests were tried out to 100 college 
first-year students. The form that showed the highest K-R 21 reliability estimate 
was selected for the final use, which is shown in Appendix A. 

The essay topic, “Plans for College Life,” was chosen since it was related to 
the students' general experience, motivated the students to write about, and was 
likely to elicit a variety of linguistic structures within the students' previously 
learned lexical and syntactical knowledge. The topic did not require 
argumentation or definition of abstract concepts. The students were expected to 
write about one page on the given topic. 

Procedure 

The cloze and essay tests were administered to four English classes of 129 
students in March, 1995 with the cooperation of the English instructors. The 
cloze test was timed to take twenty minutes and the essay test was timed to take 
twenty-five minutes. The direction for taking the cloze test and scoring method 
were read aloud to the students as they read them silently. A sample test was 
used to orient the students to the testing format. The direction and sample test 
are shown in Appendix B. 

After the collection of all cloze tests, the essay test was administered. The 
instruction, topic, and scoring criteria of the essay test were also read aloud to 
the students as they read them silently. The students were given a blank page of 
paper to write the essay on. The instructions for taking the essay test and criteria 
by which the essays were graded are shown in Appendix C. 

Scoring of the tests 

The cloze tests were scored twice by both exact-word method, which counts 
only the same words as in the original passage as correct, and acceptable-word 
method, which counts any contextually appropriate words as correct as well as 
the words in the original passage. The grading of the cloze tests was done by the 
author and a native speaker. The author graded the cloze tests by the exact-word 
method and the native speaker did by the acceptable-word method. The criteria for 
an acceptable response were taken from the previous research (Stubbs and 
Tucker, 1974). 

The essay tests were graded independently by three native speakers. The graders 
used a nine-point Likert-type scale holistically with the consideration of three 
criteria. The nine-point scale and three criteria taken in grading the essays were 
from Reid and O’Brien (1981). The nine-point scale, going from 1 (poor) to 9 
(excellent), has five main levels: poor, fair, average, good, and excellent. The 
three criteria are: 1) English structure (the use of grammar and mechanics), 2) 
content (the quantity and quality of thought and diction), and 3) organization (the 
process of thought and support). 
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Prior to the actual evaluation of the essays, a standardization process was made 
to ensure the overall reliability of the holistic essay evaluation, which was a 
grader training session. The standardization process consisted of the steps taken 
in Reid and O'Brien (1981). First, the principles of holistic grading were 
explained and discussed by the author. Then, the description of the scoring 
criteria was distributed, read, and discussed. Finally, the graders were each given a 
group of three writing samples and asked to evaluate the samples in terms of the 
criteria. When each grader had finished, discussion of the scores followed in an 
attempt to arrive at a near unanimous rating (within one point). This practice 
session was repeated three times to refine the perceptions of the graders. 

Data analysis 

Statistical analysis was performed in three ways. First, descriptive statistics 
such as means, standard deviations, and reliability estimates of the two tests was 
calculated. For the cloze test, the K-R 20 method was used to determine internal 
consistency reliability. The interrater reliability for the essays rated by all three 
graders was calculated following the steps suggested in Perkins (1986). First, the 
evaluations of all raters were intercorrelated. An average of all three correlation 
coefficients was derived. To derive the average correlation coefficient, the Fisher 
Z transformation was used because the correlation coefficient was on a non-linear 
scale. Then, the average correlation coefficient was adjusted using the Spearman- 
Brown Prophecy formula to make the final reliability estimate reflect the number 
of graders. 

Second, the transformation of raw scores of the two tests was made for the 
purpose of easy comparisons; the maximum raw scores for the tests were 30 for 
the cloze test and 9 for the essay test. Raw cloze scores were recorded and 
multiplied by 3.3 to obtain percent scores for total cloze performance. Raw essay 
scores given by the three graders were averaged and then the average scores were 
multiplied by 1 1. 1 to obtain percent scores for total essay performance. Finally, 
the Pearson’s product-moment correlation was performed to see how the two 
tests correlated with each other. It indicated the degree of commonality among 
the tests. 

RESULTS 

The findings on basic information of the two tests, such as means, standard 
deviations, and reliability estimates are first presented. Then, the result of the 
correlational analysis is presented. 

Table 1 presents the mean, standard deviation, and K-R 20 reliability estimate of 
the raw cloze test scores. 
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Table 1. 

Mean, standard deviation, and reliability of the raw cloze scores (n=129) 



Scoring Method 


Mean 


SD 


Reliability 


Exact-word 


10.21 


3.88 


.73 


Acceptable word 


13.42 


4.75 


.78 



* The maximum cloze score was 30. 



Table 1 shows that the cloze mean score of the present study by the exact-word 
scoring method was relatively high in comparison with the cloze mean scores 
obtained in previous research. In Fotos’s (1991) study, the mean score of was 12 
out of total score of 50. In Brown’s (1993) research, the means of fifty cloze 
tests ranged from 1.02 to 9.92 out of total score of 30, though his "natural” 
cloze tests were found to be more difficult than typical cloze tests. 

The mean score by the acceptable-word scoring method was higher than the 
mean score by the exact-word scoring method. The difference of the mean scores 
between the two methods was 3.21. In other words, the mean score by the 
acceptable-answer scoring method was about 10% higher than the mean score by 
the exact-answer scoring method. 

The cloze test reliability was estimated using the K-R 20 method. The K-R 21 
method used in previous research (Fotos, 1991; Hanania and Shikhani, 1986) 
was criticized for its underestimate or sometimes overestimate of reliability in 
spite of the advantage of easier calculation (Brown, 1993) and thus avoided in the 
data analysis of the main cloze test; the assumption of the K-R 21 procedure that 
all of the items are of equal difficulty is rarely met on real tests. The reliability 
estimates were fairly high, which means the cloze test is reasonably reliable to 
produce consistent results. 

The correlation coefficient between the exact-word and acceptable-word scoring 
methods was .94, which was significant (p < .01). The coefficient of 
determination, the value obtained by squaring the correlation coefficient, 
indicates the percentage of overlapping variance between the two scoring 
methods. The .88 coefficient of determination indicates the two methods were 
strongly related to each other. 

Table 2 presents the mean, standard deviation, and interrater reliability estimate 
of the raw essay test scores. Table 2 shows that students’ writing proficiency 
was slightly below the average, which was 5 on the nine-point scale going from 
1 (poor) to 9 (excellent). As the small value of standard deviation indicated, most 
essays (about 60% of the total essays) were in the category of 4 to 6, which were 
below average, average, and above average. The high .89 interrater reliability 
estimate of the essay tests calculated using the average of intercorrelation 
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coefficients and the Spearman-Brown Prophecy formula confirmed that holistic 
evaluations of essay tests are reliable (Flahive and Snow, 1980) as well as valid. 

Table 2. 

Mean, standard deviation, and reliability of the raw essay scores (n= 129) 



Mean 


SD 


Reliability 


4.22 


1.23 


.89 



* The maximum score was 9. 



Finally, in Table 3 are listed transformed scores of the two tests from raw to 
percent correct in order to make easier comparisons among the two tests. 

Table 3. 

Means and standard deviations of the percent correct scores of the cloze and essay 

tests (n=129) 



Test 


Mean 


Standard Deviation 


Cloze (Exact-word) 


33.69 


12.81 


Cloze (Acceptable word) 


44.28 


15.68 


Essay 


46.87 


13.62 



The maximum score of each test was 100. 



The relationships between the essay and cloze scores 
Table 5 presents the relationship between the cloze scores and the essay scores, 
which was expressed by the Pearson’s product-moment correlation coefficients. 

Table 5. 

The relationships among the cloze and essay (n=105) 



Paairs of Tests 


r 


Essay and Cloze (Exact- word) 


.67 


Essay and Cloze (Acceptable word) 


.64 


Both correlations were significant at pc. 01 
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The correlations indicate a degree of commonality between the tests and 
confirm their validity as tests of language proficiency. The .67 correlation 
coefficient between the essay and cloze scores is noticeable in that lower 
correlation coefficients between essay and cloze scores were found in previous 
studies: .51 from Hanania and Shikhani’s (1986) samples of college applicants 
and .46 from Fotos’s (1991) samples of college freshmen. Besides, considering 
that the students in this study were restricted samples in terms of their college 
entrance English test scores, the high correlation coefficient was remarkable; 
lower correlation coefficients are expected for a truncated sample. 

DISCUSSION 

This section discusses the results of the data analyses in light of possible 
explanations coming from the existing literature. 

Descriptive statistics 

The cloze mean score of the present study is not in line with the mean scores 
shown in previous studies (Chavez-Oller et al., 1985; Fotos, 1991). Fotos’s 
(1991) conclusion that “cloze test mean scores are rather low regardless of the 
institution, the language tested, or the type of students, especially when exact 
scoring is used” (p. 329) does not fit in the present research. The higher cloze 
mean score of the present study appears to result from the analysis of readability 
level of the participants of this study prior to the selection of the possible cloze 
passages. Unlike the previous research, this study used Dale and Chall’s (1948) 
readability formula to decide the students’ reading level correctly. The evaluation 
of the students’ reading ability was found to be the readability formula score of 
6.64, indicating the reading level of seventh to eighth graders in the U.S.A. This 
evaluation on the basis of a scientific calculation led to a more correct selection 
of cloze passages than that based on testers' subjective judgments or random 
choices made in the previous studies. 

Besides, the revision of the cloze passages made after a pilot study and the 
final selection of the cloze test made after a pretest seem to contribute to the 
higher cloze mean score. The importance of careful preparation and selection of 
cloze tests emphasized in Fotos’s (1991) study is again confirmed. This 
confirmation is in line with a study by Brown (1993) in which the conclusion 
was drawn that “tester/researcher intervention is necessary for developing sound 
cloze test” (p. 1 10). In fact, a tester or a researcher’s cloze selection decision on 
the basis of the most desirable descriptive characteristics such as reliability 
statistics makes cloze tests reliable and valid tests of overall second or foreign 
language proficiency. However, since the variation in the effectiveness of cloze 
tests has been found to be considerable (Brown, 1993), the view of cloze tests as 
an automatic indicator of second or foreign language proficiency is not advocated 
in this study. 
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The .73 (by the exact-word method) and .78 (by the acceptable-word method) 
reliability estimates of the cloze test in this study are noticeable considering that 
homogeneous samples in ability levels are often associated with low reliability 
estimates (Brown, 1993). Despite the restricted range of the cloze scores found in 
the homogeneous group of the subjects in this study, the fairly high reliability 
estimates are reported indicating the cloze test is reasonably sound. 

The strong relationship between the two scoring methods indicated by the 
correlation coefficient confirms the findings of the previous research (Oiler, 
1973; Stubbs and Tucker, 1974) and suggests the easier exact-word scoring 
method could be appropriate for use by nonnative speaker teachers. Another 
support for the use of the exact-word scoring method, which has been criticized 
for discouragingly yielding low scores to the cloze test takers, comes from the 
about 10% difference of the mean scores between the two scoring methods found 
in this study. The difference of the mean is quite small in comparison with 
Brown’s (1980) study in which the mean by the acceptable-answer scoring 
method was 7 1 % higher than the mean by the exact-answer scoring method. 

The near-average mean score and the small value of the standard deviation for 
the essay test seem to show that the subjects in this study are homogeneous 
with regard to English writing proficiency. It should, however, be noted that just 
one sample of essay may be insufficient to evaluate the writer’s composition 
ability (Henning, 1987). In fact, after finishing the essay scoring, one of the 
essay graders mentioned that nearly all essays were similar in terms of the 
amount of writing and writing style as well as the content of writing. Thus, 
rather than single compositions often criticized for unreliable representations of 
writing proficiency, several writing samples whose topics are various are 
recommended as a more correct measure of writing (Henning, 1987). 

The .89 interrater reliability estimate for the essay scores in this study 
confirms holistic evaluations can be reliable. The subjectivity involved in 
holistic scoring due to bias, fatigue, and shifting standards from one paper to the 
next is reduced by several techniques: establishment of criteria to focus graders’ 
attention on significant aspects of the writing, training of graders until they 
achieve close agreement in the assessment of the same paper, and three 
independent ratings of each essay. 

Validity of the cloze test as a writing measure 

The observed high correlation between the essay and cloze tests reflects the 
concurrent validity of the cloze test as a writing measure. The common 
integrative nature between the essay and cloze tests is proved. This means both 
tests require students to draw upon several language skills, which are production 
of language rather than recognition of correct items. Hanania and Shikhani 
(1986) and Fotos (1991) viewed the commonality between the two integrative 
tests is related to the testing of higher-order language abilities. 




10 



Cloze Test 



65 



Limitations 

There are some limitations in this study. First, in spite of the efforts put in 
the design of the cloze test used in this study, the cloze test has not resolved the 
problems of generalizability and the equivalence of items across different forms. 
This situation is related to the validity issue. Considering the remark that ”... 
validity is related more to the individual items of the test than to general 
variables, such as rate of systematic deletion ...” (p. 108, Hanania and Shikhani), 
the cloze test needs the identification of item characteristics for a more suitable 
test format. 

Second, some study (Stansfield and Hansen, 1983) shows that students’ field 
independence has a consistent relationship with cloze performance. The 
correlation found between field independence and performance on cloze tests 
seems to indicate a cognitive style bias operates in cloze solutions. On the other 
hand, test anxiety was found to affect students’ cloze performance (Oh, 1992; 
Shohamy, 1982). Considering that affective factors have a strong impact on 
students’ performance in testing and learning, the examination of the effect of 
test anxiety on cloze performance should be needed in research on cloze tests. 
Thus, certain nonlinguistic characteristics, which have not been considered in 
this study, are worthy of further investigation and interpretation. 

Finally, the evaluation of a student’ writing proficiency through an essay 
sample has been controversial, especially, when holistic scoring is used. Single 
compositions are said to be unreliable representations of writing proficiency 
(Arthur, 1979). Thus, to reduce the threats to reliability, the elicitation of 
multiple writing samples is recommended to control for the fact that writing 
proficiency may vary with topic (Henning, 1987). With more reliable 
evaluations of students’ writing proficiency, the relationship between cloze and 
essay scores would be accurately revealed. 

Implications 

The result of this study has both theoretical and practical implications. 
Theoretically speaking, the results of this study may be an answer to the 
question of what underlying factors are involved in cloze test performance. "Hie 
strong relationship between the cloze and essay tests confirms writing 
proficiency may be an important factor to explain cloze performance. Further 
investigation of the factors to influence cloze performance, both linguistic and 
nonlinguistic, may lend insights into how such factors operate in cloze 
solutions. 

Practically speaking, the confirmation of the cloze procedure as a valid 
language proficiency test suggests the use of the procedure beyond a testing 
format. It can be used as an effective teaching device. Hinofotis (1987) suggests 
some variations in the cloze format suited for classroom use such as use of the 
cloze format in an exercise in listening comprehension. Such a modified use of 
the cloze procedure is found in other research (Buck, 1988) and indicates 
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individual teachers can vary the cloze format to suit their instructional needs in 
the classroom. 

Conclusion 

This study aims to determine whether cloze tests can measure Korean students’ 
English writing proficiency. Result supports the view that cloze tests can be 
used as an integrative measure of writing proficiency. However, for the effective 
and accurate use of cloze tests, careful preparation of cloze tests should be done. 
In addition, nonlinguistic factors such as cognitive and affective style bias should 
be considered in the interpretation of cloze test scores. 
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APPENDIX A 

Cloze test 

As the head chef of a 120-seat formal Northern Italian restaurant, I have 
cooked many beautiful and delicious meals. The most important critics, our 
customers, have for four years in a row named us their favorite restaurant in this 
city. All of the hard work has paid off. 

This career of mine was not (1) with any foresight at all. 

Rather, (2) kind of snuck up on me (3) blossomed 

unexpectedly. 

The beginnings of my (4) were quite typical: I was out 

(5) college, broke, and in debt. When (6) set out to look 

for a (7) , my criteria were simply to find (8) that I 

enjoyed, and to use (9) natural ability. I decided to try 

GO) • 
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At first, I wondered whether I (1 1) made a wise choice. 

My first (12) position exposed me to all of (13) evils 

of the restaurant business. There (14) the owners who cared only 

about (15) bottom line, and the managers who (16) 

nothing about food and who cared (17) less. 

1 wanted a job where (18) cooking was taking place. I 

looked (19) a place where quality products would (20) 

handled professionally, and where I would (21) able to contribute 

what 1 had (22) offer. 

So, eight years ago, 1 (23) an apprentice here at 

Giovanni’s. Working (24) this kitchen proved to be a 

(25) “education” for me. 1 gradually became (26) of how 

temperature, cooking times, and (27) affected the look, texture, and 

flavor (28) the food 1 was preparing. 

On (29) down side, the hours were murder. 

(30) kitchen was as hot as a furnace. I worked under a disagreeable 

chef. 

This man eventually left, and that was the turning point for me. 
Although I had very little experience, almost no trained help, and a hot summer 
ahead, I took over as the new chef. 

Answers for cloze test 

1) planned 2) it 3) and 4) career 5) of 6)1 7) job 8) something 9) my 
10) cooking 11) had 12) cooking 13) the 14) were 15) the 16) knew 17) 
even 18) real 19) for 20) be 21) be 22) to 23) became 24) in 25) 
good 26) aware 27) ingredients 28) of 29) the 30) The 



APPENDIX B 

Directions for Taking Cloze Tests 

At the bottom of this page is a sample of a new kind of test. The test is 
made by copying a few paragraphs from a book. Every seventh word was left out 
of the paragraphs, and thirty blank spaces were put where the words were taken 
out. 

Your job will be to guess what word was left out of each space and to 
write that word in that space. It will help you in taking the test if you remember 
these things: 

1. Write only one word in each blank. 

2. Try to fill every blank. Don't be afraid to guess. 

3. You may skip hard blanks and come back to them when you have finished. 
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4. Wrong spelling will not count against you if we can tell what word you 
meant. 

5. All blanks can be answered with ordinary words. 

Sample Test 

Below is a sample of the test to help you to be familiar to the main 
test. Fill each blank with the word you think was taken out. You may check 
your paper when you finish it by looking at the answers which are written at the 
bottom of the page. 

What is a friend? More than (1) father, more than a brother: A 

(2) companion. With him, you can conquer (3) impossible. 

l)a 2) traveling 3) the 

APPENDIX C 

Instruction 

This is a test of your ability to write an essay in English. When I give 
you the test paper, you will have twenty five minutes to write an about one page 
essay. The topic you are write about is stated on the test. You may take a few 
minutes, if you want, at the beginning of the test, to make notes or an outline 
for your essay on the back of the paper you receive. If you do not have enough 
space on the front of the page, you may continue on the back. 

Criteria of evaluation 

Your essay will be evaluated as to your ability to communicate effectively in 
writing. We will look at three qualities in particular: English structure, content, 
and organization. 
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